Semantic image understanding : from pixel to word

The aim of semantic image understanding is to reveal the semantic meaning behind the image pixel. We categorize semantic image understanding into two broad categories: pixel-level and image-level semantic image understanding. While pixel-level image understanding aims to obtain the semantic meaning of each pixel, image-level understanding aims to obtain the semantic meaning of the whole image, and both levels involve feature extraction and combination. In this thesis, we study semantic image understanding and have made following novel contributions: We investigated the utility of Multiple Kernel Learning (MKL) for feature combination. We introduced the concept of kernel histogram. We observed that the kernel histograms of different features are usually very different, and argued that traditional MKL’s linear kernel combination strategy is not particularly meaningful. Then we proposed the concept of Relative Kernel Distribution Invariance (RKDI) for kernel combination, and have developed a very simple histogram matching technique to achieve RKDI by transforming different kernel histograms to a canonical histogram. We have also developed two kinds of measure for automatically choosing the canonical histogram. Extensive experiments on various computer vision and machine learning datasets have shown that calibrating the kernels to a canonical histogram before they are linearly combined can always achieve a performance gain over state of the art MKL methods. For the problem of understanding image at the pixel level, we advocate the segment-then-recognize strategy. We have developed a new framework which tries to integrate semantic segmentation with low-level segmentation by introducing a semantic feature feedback mechanism. Experiments on two wellknown datasets have confirmed that our new segmentation method can indeed produce regions that are more object-consistent. Besides this, we have also developed a novel idea trying to integrate semantic segmentation with interactive segmentation. We treat the semantic segmentation module as an unreliable teacher which automatically generates tokens to guide the interactive segmentation module. Some qualitative results have shown the promise of this approach.